Television broadcasters, as well as cable system and network operators, including Internet servers, and other video delivery systems which use video servers to store and play out digital video signals, have very stringent requirements for reliability since in most cases video stream interrupts cannot be tolerated. To meet such reliability standards, video servers are often employed in pairs where one of the servers serves as a "hot standby" in case the primary video server experiences a failure.
Video servers typically store large amounts of video (also referred to as "video content" or "content") on DASD (direct access storage devices), and when configured redundantly, the primary and secondary systems have the exact same video content. In addition, both servers ideally will play identical video out of respective video ports, although only the output of the primary server actually gets broadcast. In this manner, the secondary server may function in a standby mode and can be switched "on-line" immediately.
Although video servers may store many hours of high quality video content, in practice, a cable company or a television studio must continuously delete old videos from the servers and copy or stage in new videos that are scheduled to play in the near future. That practice is called "content management" and it is an automated process that takes place concurrently while videos are being played. When new videos are staged, they are staged to both the primary and the secondary servers simultaneously so that the two servers continue to be "mirror images" of each other.
If the secondary server shuts down due to scheduled or unscheduled maintenance for any length of time, its DASD may no longer be a mirror image of the primary server because the server was "off-line" while content was being deleted from or staged to the primary server. Because of this, the secondary server's ability to function as a "hot standby" is degraded and the server administrator must manually restore the missing content and delete the extraneous videos. In the worst case, videos that are currently played or queued may be missing from the secondary server which means that one or more channels would be off the air if a server "switch" were needed and this would result in a loss of revenue.
When the content of the redundant or secondary server is no longer in synch with the primary server, the server administrator must first determine which videos are needed soonest and stage those videos in as soon as possible. In order to make room for the new content, the administrator may first need to delete some "unneeded" content. For example, one or more of the videos may have played on the primary server while the secondary server was out of service and those videos will not be needed when the secondary server is re-synched with the primary server. Another factor the administrator must: consider is the current processing and resource load of the video servers. If the missing videos are staged in at a high rate of transfer, the normally scheduled staging of content may be slowed down and also the quality of the videos being played may be degraded. Staging and playing of videos consumes a certain amount of CPU (Central Processing Unit) and data bus bandwidth and the administrator can not allow any degradation of the video quality.
The problem is further complicated since the CPU and data bus utilization is very dynamic and the administrator has no way of measuring the current load and predicting load changes in the near future. Also, the administrator may be located across the country in which case the administrator may not have access to the current schedule and will not be able to prioritize the order of staging videos.
Another serious problem is that the administrator has no way to start playing a missing video after it has been staged into the secondary server. If the missing video is a two hour movie for example, the server output port will be idle for up to two hours until there is a command to play the queued video. Even if the administrator could play the video, the administrator would first have to query the current location within the movie being played by the primary server, and then "fast forward" to that point. This cannot be done manually with any kind of accuracy, even with highly trained personnel.
More specifically, in the video broadcast industry, standards require the start of video playout to be "frame accurate". This frame accuracy is achieved by the use of master automation control computers to control video devices such as video tape recorders and digital video servers. Videos are played out according to the air times specified in a playlist and a minimum of two server commands are required to play a video, i.e. a "queue" command and a "play" command. Implicit in the queue command is the starting point of the video which is assumed to be the first frame. To start a video at some time offset into the video, a "queue with data" command is used which specifies the starting offset in hours, minutes, seconds and frames. As hereinbefore discussed, broadcast studios typically employ a second video server as a hot standby in case the primary server fails. By executing identical playlists in a master control automation computer, both servers stream the same videos out of the same ports at precisely the same play offset. When the primary server fails, the secondary server is promptly switched on-air and now becomes the primary or "on-air" server.
However, while the secondary server is down, all master automation commands sent addressed to the secondary server fail. When the failed server is repaired and brought back on-line, the stream outputs remain black because master control automation is not designed to restore service. The secondary server will not be synched up with the primary server until the next queue and play commands are executed for each port which may be anywhere from 2 seconds to two hours in the future. To restore service promptly, a broadcast engineer would have to take manual control of the secondary server's playlist and execute a "Queue With Data" command using a start offset that was manually calculated, followed by a "Play" command. This must be done for each server port with an active stream. This manual restoration procedure is difficult and error prone for even a skilled broadcast engineer and to sync up the streams of the secondary server with the streams of the primary server within even 2 seconds of each other is adventitious, and even that is far less accurate than the desired frame specific standard which is sought.
Accordingly, there is a need for an enhanced method and processing apparatus which are effective to synchronize primary and secondary video servers in a video transmission system to a high degree of accuracy with respect to video content and video streaming functions of the transmission system.